{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 理解反向传播算法\n",
    "\n",
    "此节内容来源于唐一旦老师的视频 [徒手实现反向传播算法--算法细节与代码实现](https://www.bilibili.com/video/BV12b421v7qq) 。\n",
    "\n",
    "主要是模拟 Pytorch 中的计算图来理解反向传播算法（链式求导），在此之前我们需要理解几个概念：\n",
    "\n",
    "- 向前传播：输入数据，得到预测结果。\n",
    "- 反向传播：计算梯度（在 Pytorch 中这种算法叫 Autograd）\n",
    "- 向后传播：计算梯度、更新参数"
   ],
   "id": "214a7195e2e08455"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T10:13:51.379606Z",
     "start_time": "2025-08-11T10:13:46.936797Z"
    }
   },
   "cell_type": "code",
   "source": "!pip install graphviz",
   "id": "a41e7fe7655bccf8",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting graphviz\n",
      "  Using cached graphviz-0.21-py3-none-any.whl.metadata (12 kB)\n",
      "Using cached graphviz-0.21-py3-none-any.whl (47 kB)\n",
      "Installing collected packages: graphviz\n",
      "Successfully installed graphviz-0.21\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 24.0 -> 25.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "execution_count": 8
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T12:54:37.984734Z",
     "start_time": "2025-08-11T12:54:37.978704Z"
    }
   },
   "cell_type": "code",
   "source": [
    "class ScalarTmp:\n",
    "\n",
    "    def __init__(self, value, prevs=[], op=None, label=''):\n",
    "        \"\"\"\n",
    "        定义一个自定义的数据类型\n",
    "\n",
    "        :param value: 参与运算的结果\n",
    "        :param prevs: 参与运算的前序节点\n",
    "        :param op: 进行的操作\n",
    "        :param label: 数字的标签\n",
    "        \"\"\"\n",
    "\n",
    "        self.value = value\n",
    "        self.prevs = prevs\n",
    "        self.op = op\n",
    "        self.label = label\n",
    "        self.grad = 0.0  # 全局偏导：计算图的顶点对该节点的偏导数\n",
    "        self.grad_wrt = {}  # 该节点对直接相连的前序节点的偏导数\n",
    "\n",
    "    # 下面这些都属于 Python 的基本语法知识了\n",
    "    def __add__(self, other):\n",
    "        value = self.value + other.value\n",
    "        prevs = [self, other]\n",
    "        label = f\"({self.label}) + {other.label}\"\n",
    "        output = ScalarTmp(value, prevs, '+', label)\n",
    "        output.grad_wrt[self] = 1\n",
    "        output.grad_wrt[other] = 1\n",
    "        return output\n",
    "\n",
    "\n",
    "    def __mul__(self, other):\n",
    "        value = self.value * other.value\n",
    "        prevs = [self, other]\n",
    "        label = f\"({self.label}) * {other.label}\"\n",
    "        output = ScalarTmp(value, prevs, '*', label)\n",
    "        output.grad_wrt[self] = other.value\n",
    "        output.grad_wrt[other] = self.value\n",
    "        return output\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f\"{self.value} | {self.op} | {self.label}\""
   ],
   "id": "56117d809f67a5be",
   "outputs": [],
   "execution_count": 24
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T13:47:28.099808Z",
     "start_time": "2025-08-11T13:47:28.095766Z"
    }
   },
   "cell_type": "code",
   "source": [
    "a = ScalarTmp(1.0, label='a')\n",
    "b = ScalarTmp(2.0, label='b')\n",
    "c = ScalarTmp(4.0, label='c')\n",
    "d = a + b\n",
    "e = a * c\n",
    "f = d * e\n",
    "print(f)"
   ],
   "id": "a9d103a9bb4921a0",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.0 | * | ((a) + b) * (a) * c\n"
     ]
    }
   ],
   "execution_count": 66
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "偏导数的链式法则：\n",
    "$$\n",
    "\\frac{\\partial y}{\\partial a} = \\frac{\\partial y}{\\partial o_1} * \\frac{\\partial o_1}{\\partial a} + \\cdots + \\frac{\\partial y}{\\partial o_n} * \\frac{\\partial o_n}{\\partial a} = \\sum_{i=1}^{n} \\frac{\\partial y}{\\partial o_i} * \\frac{\\partial o_i}{\\partial a}\n",
    "$$"
   ],
   "id": "eabfbb9b1860b7ea"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T13:42:27.901788Z",
     "start_time": "2025-08-11T13:42:27.896478Z"
    }
   },
   "cell_type": "code",
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "def _trace(root):\n",
    "    # 遍历计算图中的所有点和边\n",
    "    nodes, edges = set(), set()\n",
    "    def _build(v):\n",
    "        if v not in nodes:\n",
    "            nodes.add(v)\n",
    "            for prev in v.prevs:\n",
    "                edges.add((prev, v))\n",
    "                _build(prev)\n",
    "    _build(root)\n",
    "    return nodes, edges\n",
    "\n",
    "\n",
    "def draw_graph(root, direction='forward'):\n",
    "    nodes, edges = _trace(root)\n",
    "    rankdir = 'BT' if direction == 'forward' else 'TB'\n",
    "    graph = Digraph(format='svg', graph_attr={'rankdir': rankdir})\n",
    "    # 画点\n",
    "    for node in nodes:\n",
    "        label = node.label if node.op is None else node.op\n",
    "        node_attr = f'{{ grad={node.grad} | value={node.value:.2f} | {label}}}'\n",
    "        uid = str(id(node))\n",
    "        graph.node(name=uid, label=node_attr, shape='record')\n",
    "    # 画边\n",
    "    for edge in edges:\n",
    "        id1 = str(id(edge[0]))\n",
    "        id2 = str(id(edge[1]))\n",
    "        graph.edge(id1, id2)\n",
    "    return graph"
   ],
   "id": "24ea3fc9dbbaedfe",
   "outputs": [],
   "execution_count": 49
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T13:42:30.827713Z",
     "start_time": "2025-08-11T13:42:29.815023Z"
    }
   },
   "cell_type": "code",
   "source": "draw_graph(f)",
   "id": "48d68c7858278226",
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 12.1.0 (20240811.2233)\n -->\n<!-- Pages: 1 -->\n<svg width=\"278pt\" height=\"304pt\"\n viewBox=\"0.00 0.00 277.50 303.50\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 299.5)\">\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-299.5 273.5,-299.5 273.5,4 -4,4\"/>\n<!-- 1895635180112 -->\n<g id=\"node1\" class=\"node\">\n<title>1895635180112</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"144,-111 144,-184.5 221.5,-184.5 221.5,-111 144,-111\"/>\n<text text-anchor=\"middle\" x=\"182.75\" y=\"-167.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=0.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"144,-160 221.5,-160\"/>\n<text text-anchor=\"middle\" x=\"182.75\" y=\"-142.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=3.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"144,-135.5 221.5,-135.5\"/>\n<text text-anchor=\"middle\" x=\"182.75\" y=\"-118.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">+</text>\n</g>\n<!-- 1895635184272 -->\n<g id=\"node3\" class=\"node\">\n<title>1895635184272</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"92.62,-221.5 92.62,-295 176.88,-295 176.88,-221.5 92.62,-221.5\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-277.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=0.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"92.62,-270.5 176.88,-270.5\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-253.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=12.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"92.62,-246 176.88,-246\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-228.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">*</text>\n</g>\n<!-- 1895635180112&#45;&gt;1895635184272 -->\n<g id=\"edge1\" class=\"edge\">\n<title>1895635180112&#45;&gt;1895635184272</title>\n<path fill=\"none\" stroke=\"black\" d=\"M166.77,-184.88C163.08,-193.21 159.11,-202.18 155.27,-210.87\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"152.12,-209.33 151.28,-219.89 158.52,-212.17 152.12,-209.33\"/>\n</g>\n<!-- 1895633835088 -->\n<g id=\"node2\" class=\"node\">\n<title>1895633835088</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-74 77.5,-74 77.5,-0.5 0,-0.5\"/>\n<text text-anchor=\"middle\" x=\"38.75\" y=\"-56.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=0.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"0,-49.5 77.5,-49.5\"/>\n<text text-anchor=\"middle\" x=\"38.75\" y=\"-32.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=4.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"0,-25 77.5,-25\"/>\n<text text-anchor=\"middle\" x=\"38.75\" y=\"-7.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">c</text>\n</g>\n<!-- 1895635183440 -->\n<g id=\"node5\" class=\"node\">\n<title>1895635183440</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"48,-111 48,-184.5 125.5,-184.5 125.5,-111 48,-111\"/>\n<text text-anchor=\"middle\" x=\"86.75\" y=\"-167.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=0.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"48,-160 125.5,-160\"/>\n<text text-anchor=\"middle\" x=\"86.75\" y=\"-142.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=4.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"48,-135.5 125.5,-135.5\"/>\n<text text-anchor=\"middle\" x=\"86.75\" y=\"-118.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">*</text>\n</g>\n<!-- 1895633835088&#45;&gt;1895635183440 -->\n<g id=\"edge5\" class=\"edge\">\n<title>1895633835088&#45;&gt;1895635183440</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.73,-74.38C58.42,-82.71 62.39,-91.68 66.23,-100.37\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"62.98,-101.67 70.22,-109.39 69.38,-98.83 62.98,-101.67\"/>\n</g>\n<!-- 1895633837776 -->\n<g id=\"node4\" class=\"node\">\n<title>1895633837776</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"192,-0.5 192,-74 269.5,-74 269.5,-0.5 192,-0.5\"/>\n<text text-anchor=\"middle\" x=\"230.75\" y=\"-56.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=0.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"192,-49.5 269.5,-49.5\"/>\n<text text-anchor=\"middle\" x=\"230.75\" y=\"-32.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=2.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"192,-25 269.5,-25\"/>\n<text text-anchor=\"middle\" x=\"230.75\" y=\"-7.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">b</text>\n</g>\n<!-- 1895633837776&#45;&gt;1895635180112 -->\n<g id=\"edge3\" class=\"edge\">\n<title>1895633837776&#45;&gt;1895635180112</title>\n<path fill=\"none\" stroke=\"black\" d=\"M214.77,-74.38C211.08,-82.71 207.11,-91.68 203.27,-100.37\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"200.12,-98.83 199.28,-109.39 206.52,-101.67 200.12,-98.83\"/>\n</g>\n<!-- 1895635183440&#45;&gt;1895635184272 -->\n<g id=\"edge4\" class=\"edge\">\n<title>1895635183440&#45;&gt;1895635184272</title>\n<path fill=\"none\" stroke=\"black\" d=\"M102.73,-184.88C106.42,-193.21 110.39,-202.18 114.23,-210.87\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"110.98,-212.17 118.22,-219.89 117.38,-209.33 110.98,-212.17\"/>\n</g>\n<!-- 1895633837968 -->\n<g id=\"node6\" class=\"node\">\n<title>1895633837968</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"96,-0.5 96,-74 173.5,-74 173.5,-0.5 96,-0.5\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-56.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=0.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"96,-49.5 173.5,-49.5\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-32.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=1.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"96,-25 173.5,-25\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-7.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">a</text>\n</g>\n<!-- 1895633837968&#45;&gt;1895635180112 -->\n<g id=\"edge2\" class=\"edge\">\n<title>1895633837968&#45;&gt;1895635180112</title>\n<path fill=\"none\" stroke=\"black\" d=\"M150.73,-74.38C154.42,-82.71 158.39,-91.68 162.23,-100.37\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"158.98,-101.67 166.22,-109.39 165.38,-98.83 158.98,-101.67\"/>\n</g>\n<!-- 1895633837968&#45;&gt;1895635183440 -->\n<g id=\"edge6\" class=\"edge\">\n<title>1895633837968&#45;&gt;1895635183440</title>\n<path fill=\"none\" stroke=\"black\" d=\"M118.77,-74.38C115.08,-82.71 111.11,-91.68 107.27,-100.37\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"104.12,-98.83 103.28,-109.39 110.52,-101.67 104.12,-98.83\"/>\n</g>\n</g>\n</svg>\n",
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1b95c972e50>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 50
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T13:42:33.759233Z",
     "start_time": "2025-08-11T13:42:33.755726Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 拓扑排序\n",
    "def _top_order(root):\n",
    "    # 深度优先搜索\n",
    "    ordered, visited = [], set()\n",
    "    def _add_prevs(node):\n",
    "        if node not in visited:\n",
    "            visited.add(node)\n",
    "            for prev in node.prevs:\n",
    "                _add_prevs(prev)\n",
    "            ordered.append(node)\n",
    "    _add_prevs(root)\n",
    "    return ordered"
   ],
   "id": "4caeaa382108850f",
   "outputs": [],
   "execution_count": 51
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T13:42:34.979883Z",
     "start_time": "2025-08-11T13:42:34.975376Z"
    }
   },
   "cell_type": "code",
   "source": "_top_order(f)",
   "id": "10d368d6c7fcf348",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1.0 | None | a,\n",
       " 2.0 | None | b,\n",
       " 3.0 | + | (a) + b,\n",
       " 4.0 | None | c,\n",
       " 4.0 | * | (a) * c,\n",
       " 12.0 | * | ((a) + b) * (a) * c]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 52
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T13:54:54.693973Z",
     "start_time": "2025-08-11T13:54:54.689959Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def backward(root):\n",
    "    # 定义顶点梯度\n",
    "    root.grad = 1.0\n",
    "    for node in reversed(_top_order(root)):\n",
    "        for v in node.prevs:\n",
    "            v.grad += node.grad * node.grad_wrt[v]\n",
    "    return root"
   ],
   "id": "8fd242f3583168eb",
   "outputs": [],
   "execution_count": 73
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-11T13:48:02.061652Z",
     "start_time": "2025-08-11T13:48:01.048489Z"
    }
   },
   "cell_type": "code",
   "source": [
    "backward(f)\n",
    "draw_graph(f)"
   ],
   "id": "2b581e3037349897",
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 12.1.0 (20240811.2233)\n -->\n<!-- Pages: 1 -->\n<svg width=\"278pt\" height=\"304pt\"\n viewBox=\"0.00 0.00 277.50 303.50\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 299.5)\">\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-299.5 273.5,-299.5 273.5,4 -4,4\"/>\n<!-- 1895634880016 -->\n<g id=\"node1\" class=\"node\">\n<title>1895634880016</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"92.62,-221.5 92.62,-295 176.88,-295 176.88,-221.5 92.62,-221.5\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-277.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=1.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"92.62,-270.5 176.88,-270.5\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-253.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=12.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"92.62,-246 176.88,-246\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-228.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">*</text>\n</g>\n<!-- 1895635183632 -->\n<g id=\"node2\" class=\"node\">\n<title>1895635183632</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"192,-0.5 192,-74 269.5,-74 269.5,-0.5 192,-0.5\"/>\n<text text-anchor=\"middle\" x=\"230.75\" y=\"-56.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=3.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"192,-49.5 269.5,-49.5\"/>\n<text text-anchor=\"middle\" x=\"230.75\" y=\"-32.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=4.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"192,-25 269.5,-25\"/>\n<text text-anchor=\"middle\" x=\"230.75\" y=\"-7.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">c</text>\n</g>\n<!-- 1895632295376 -->\n<g id=\"node6\" class=\"node\">\n<title>1895632295376</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"144,-111 144,-184.5 221.5,-184.5 221.5,-111 144,-111\"/>\n<text text-anchor=\"middle\" x=\"182.75\" y=\"-167.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=3.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"144,-160 221.5,-160\"/>\n<text text-anchor=\"middle\" x=\"182.75\" y=\"-142.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=4.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"144,-135.5 221.5,-135.5\"/>\n<text text-anchor=\"middle\" x=\"182.75\" y=\"-118.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">*</text>\n</g>\n<!-- 1895635183632&#45;&gt;1895632295376 -->\n<g id=\"edge4\" class=\"edge\">\n<title>1895635183632&#45;&gt;1895632295376</title>\n<path fill=\"none\" stroke=\"black\" d=\"M214.77,-74.38C211.08,-82.71 207.11,-91.68 203.27,-100.37\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"200.12,-98.83 199.28,-109.39 206.52,-101.67 200.12,-98.83\"/>\n</g>\n<!-- 1895632292432 -->\n<g id=\"node3\" class=\"node\">\n<title>1895632292432</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"48,-111 48,-184.5 125.5,-184.5 125.5,-111 48,-111\"/>\n<text text-anchor=\"middle\" x=\"86.75\" y=\"-167.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=4.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"48,-160 125.5,-160\"/>\n<text text-anchor=\"middle\" x=\"86.75\" y=\"-142.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=3.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"48,-135.5 125.5,-135.5\"/>\n<text text-anchor=\"middle\" x=\"86.75\" y=\"-118.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">+</text>\n</g>\n<!-- 1895632292432&#45;&gt;1895634880016 -->\n<g id=\"edge5\" class=\"edge\">\n<title>1895632292432&#45;&gt;1895634880016</title>\n<path fill=\"none\" stroke=\"black\" d=\"M102.73,-184.88C106.42,-193.21 110.39,-202.18 114.23,-210.87\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"110.98,-212.17 118.22,-219.89 117.38,-209.33 110.98,-212.17\"/>\n</g>\n<!-- 1895635177232 -->\n<g id=\"node4\" class=\"node\">\n<title>1895635177232</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-74 77.5,-74 77.5,-0.5 0,-0.5\"/>\n<text text-anchor=\"middle\" x=\"38.75\" y=\"-56.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=4.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"0,-49.5 77.5,-49.5\"/>\n<text text-anchor=\"middle\" x=\"38.75\" y=\"-32.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=2.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"0,-25 77.5,-25\"/>\n<text text-anchor=\"middle\" x=\"38.75\" y=\"-7.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">b</text>\n</g>\n<!-- 1895635177232&#45;&gt;1895632292432 -->\n<g id=\"edge1\" class=\"edge\">\n<title>1895635177232&#45;&gt;1895632292432</title>\n<path fill=\"none\" stroke=\"black\" d=\"M54.73,-74.38C58.42,-82.71 62.39,-91.68 66.23,-100.37\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"62.98,-101.67 70.22,-109.39 69.38,-98.83 62.98,-101.67\"/>\n</g>\n<!-- 1895635183952 -->\n<g id=\"node5\" class=\"node\">\n<title>1895635183952</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"96,-0.5 96,-74 173.5,-74 173.5,-0.5 96,-0.5\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-56.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">grad=16.0</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"96,-49.5 173.5,-49.5\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-32.2\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value=1.00</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"96,-25 173.5,-25\"/>\n<text text-anchor=\"middle\" x=\"134.75\" y=\"-7.7\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">a</text>\n</g>\n<!-- 1895635183952&#45;&gt;1895632292432 -->\n<g id=\"edge6\" class=\"edge\">\n<title>1895635183952&#45;&gt;1895632292432</title>\n<path fill=\"none\" stroke=\"black\" d=\"M118.77,-74.38C115.08,-82.71 111.11,-91.68 107.27,-100.37\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"104.12,-98.83 103.28,-109.39 110.52,-101.67 104.12,-98.83\"/>\n</g>\n<!-- 1895635183952&#45;&gt;1895632295376 -->\n<g id=\"edge3\" class=\"edge\">\n<title>1895635183952&#45;&gt;1895632295376</title>\n<path fill=\"none\" stroke=\"black\" d=\"M150.73,-74.38C154.42,-82.71 158.39,-91.68 162.23,-100.37\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"158.98,-101.67 166.22,-109.39 165.38,-98.83 158.98,-101.67\"/>\n</g>\n<!-- 1895632295376&#45;&gt;1895634880016 -->\n<g id=\"edge2\" class=\"edge\">\n<title>1895632295376&#45;&gt;1895634880016</title>\n<path fill=\"none\" stroke=\"black\" d=\"M166.77,-184.88C163.08,-193.21 159.11,-202.18 155.27,-210.87\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"152.12,-209.33 151.28,-219.89 158.52,-212.17 152.12,-209.33\"/>\n</g>\n</g>\n</svg>\n",
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1b95c808b90>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 70
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "上图中展示的是梯度传播的算法（最上面的节点是 $f$，$f$ 的左孩子是 $d$ 右孩子是 $e$），现在我来进行一个简单的推理，最终目标是要求出所有节点的梯度，而每个节点的梯度都依赖上一个节点的传播（链式求导），举个例子来说：\n",
    "\n",
    "假设我要求 $a$ 节点的梯度，我们可以给出如下的表示：\n",
    "\n",
    "$\n",
    "\\frac{\\partial f}{\\partial a} = \\frac{\\partial f}{\\partial d} * \\frac{\\partial d}{\\partial a} + \\frac{\\partial f}{\\partial e} * \\frac{\\partial e}{\\partial a}\n",
    "$\n",
    "\n",
    "要求 $a$ 对 $f$ 的求导，必须先知道 $d$ 对 $f$ 的求导，和 $e$ 对 $f$ 的求导。\n",
    "\n",
    "我们已知：\n",
    "$\n",
    "\\begin{cases}\n",
    "f = d * e \\\\\n",
    "e = a * c\n",
    "\\end{cases}\n",
    "$\n",
    "\n",
    "不难得出：\n",
    "\n",
    "$\n",
    "\\begin{cases}\n",
    "\\frac{\\partial f}{\\partial d} = e = 4 \\\\\n",
    "\\frac{\\partial f}{\\partial e} = d = 3 \\\\\n",
    "\\frac{\\partial d}{\\partial a} = 1 \\\\\n",
    "\\frac{\\partial e}{\\partial a} = c = 4 \\\\\n",
    "\\end{cases}\n",
    "$\n",
    "\n",
    "另外，$f$ 对自己的导数是 1，不难得出，$a$ 的梯度为 $1 * 4 * 1 + 1 * 3 * 4 = 16$"
   ],
   "id": "457e3f382612af3c"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
